%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%                                                               %
%                       Legal Notice                            %
%                                                               %
% This document is copyright (C) Olaf Kirch, okir@monad.swb.de  %
%                                                               %
% For copying conditions, please refer to the file COPYING.	%
%                                                               %
% The author is not liable for any damages, direct or indirect, %
% resulting from the use of information provided in this        %
% document.                                                     %
%                                                               %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% When viewing this document with vi on an 80-columns screen,   %
% do a ':se nonumber' first.                                    %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Definitions local to this chapter.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Warning! These paths occur literally in a few verbatim environments!
%%
\def\newslib{\file{/usr/lib/news}}
\def\newsbin{\file{/usr/lib/news/bin}}
\def\newsspool{\file{/var/spool/news}}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%                                                               %
%               S T A R T   O F   C H A P T E R                 %
%                                                               %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\chapter{Netnews}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\label{news}


\section{Usenet History}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% There is a good description of the history of Usenet in Brendan
%% Kehoe's ``Zen and the Art of the Internet'', a booklet published in
%% 1992. The following is a boiled-down rewording its Usenet chapter.
%% Maybe I should get Brendan's permission to quote him here.
%% Or, it might be better to cut down on that. People should read
%% Zen Zemzelves.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\label{news.history}
\index{news|(}

The idea of network news was born in 1979 when two graduate students, Tom
Truscott and Jim Ellis, thought of using UUCP to connect machines for the 
purpose of information exchange among \unix{} users. They set up
a small network of three machines in North Carolina.


Initially, traffic was handled by a number of shell scripts (later
rewritten in C), but they were never released to the public. They
were quickly replaced by ``A'' news, the first public release of news
software.


``A'' news was not designed to handle more than a few articles
per group and day. When the volume continued to grow, it was rewritten
by Mark Horton and Matt Glickman, who called it the ``B'' release
(a.k.a. Bnews). The first public release of Bnews was version~2.1
in 1982. It was expanded continuously, with several new features 
being added. Its current version is Bnews~2.11. It is slowly
becoming obsolete, with its last official maintainer having switched to
INN.


\index{Collyer, Geoff}
\index{Spencer, Henry}
\index{news!C release|see {C News}}
\index{C News}
Another rewrite was done and released in 1987 by Geoff Collyer and Henry
Spencer; this is release ``C'', or C~News. In the time following there
have been a number of patches to C~News, the most prominent being the
C~News Performance Release. On sites that carry a large number of groups,
the overhead involved in frequently invoking \prog{relaynews}, which is
responsible for dispatching incoming articles to other hosts, is
significant. The Performance Release adds an option to \prog{relaynews}
that allows to run it in \emph{daemon mode}, in which the program puts
itself in the background.


The Performance Release is the C~News version currently included in most
\linux{} releases.


\index{Network News Transfer Protocol|see {NNTP}}
\index{NNTP}
All news releases up to ``C'' are primarily targeted for UUCP networks,
although they may be used in other environments as well.  Efficient news
transfer over networks like TCP/IP, DECNet, or related requires a new
scheme. This was the reason why, in 1986, the ``Network News Transfer
Protocol'', NNTP, was introduced. It is based on network connections,
and specifies a number of commands to interactively transfer and
retrieve articles.


There are a number of NNTP-based applications available from
the Net. One of them is the \prog{nntpd} package by Brian Barber
and Phil Lapsley, which you can use, among other things, to
provides newsreading service to a number of hosts inside a local
network. \prog{nntpd} was designed to complement news packages such as
Bnews or C~News to give them NNTP features.


\index{InterNet News (INN)}
\index{INN}
A different NNTP package is INN, or Internet News. It is not merely
a front end, but a news system by its own right. It comprises a
sophisticated news relay daemon that is capable of maintaining
several concurrent NNTP links efficiently, and is therefore the
news server of choice for many Internet sites.


\section{What {\protect\em is} Usenet, Anyway?}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\label{news.usenet}
\index{news!Usenet}
\index{Usenet}
\index{Zen}

One of the most astounding facts about Usenet is that it isn't part of
any organization, or has any sort of centralized network management
authority. In fact, it's part of Usenet lore that except for a technical
description, you cannot define \emph{what} it is, you can only say what
it isn't. If you have Brendan Kehoe's excellent ``Zen and the Art of the
Internet'' (available online or through Prentice-Hall, see \cite{zen})
at hand, you will find an amusing list of Usenet's non-properties.


\index{news!exchanging}
\index{exchanging!news}
\index{news!feeding}
At the risk of sounding stupid, one might define Usenet as a
collaboration of separate sites who exchange Usenet news.  To be a
Usenet site, all you have to do is find another site Usenet site, and
strike an agreement with its owners and maintainers to exchange news
with you. Providing another site with news is also called \emph{feeding}
it, whence another common axiom of Usenet philosophy originates: ``Get a
feed and you're on it.''


\index{news!article}
The basic unit of Usenet news is the article. This is a message a user
writes and ``posts'' to the net. In order to enable news sytems to deal
with it, it is prepended with administrative information, the so-called
article header. It is very similar to the mail header format laid down
in the Internet mail standard RFC~822, in that it consists of several
lines of text, each beginning with a field name terminated by a colon,
which is followed by the field's value.\footnote{
 The format of Usenet news messages is specified in RFC~1036,
 ``Standard for interchange of USENET messages''.
}


\index{news!groups}
Articles are submitted to one or more \emph{newsgroups}.  One may
consider a newsgroup a forum for articles relating to a common topic.
All newsgroups are organized in a hierarchy, with each group's name
indicating its place in the hierarchy. This often makes it easy to see
what a group is all about. For example, anybody can see from the
newsgroup name that \ngrp{comp.os.linux.announce} is used for
announcements concerning a computer operating system named \linux.


\index{news!exchanging}
\index{exchanging!news}
\index{news!feeding}
These articles are then exchanged between all Usenet sites that are
willing to carry news from this group.  When two sites agree to exchange
news, they are free to exchange whatever newsgroups they like to, and
may even add their own local news hierarchies. For example,
\host{groucho.edu} might have a news link to \host{barnyard.edu}, which
is a major news feed, and several links to minor sites which it feeds
news. Now, Barnyard College might receive all Usenet groups, while GMU
only wants to carry a few major hierarchies like \ngrp{sci},
\ngrp{comp}, \ngrp{rec}, etc. Some of the downstream sites, say a UUCP
site called \host{brewhq}, will want to carry even fewer groups, because
they don't have the network or hardware resources. On the other hand,
\host{brewhq} might want to receive newsgroups from the \ngrp{fj}
hierarchy, which GMU doesn't carry. It therefore maintains another link
with \host{gargleblaster.com}, who carry all \ngrp{fj} groups, and feed
them to \host{ brewhq}. The news flow is shown in
figure~\ref{news.fig.article-flow}.


\begin{figure}[htbp]
\iftex
 \ifps
  \begin{center}
   \mbox{\epsffile{figures/gmu.epsf}}
  \end{center}
 \else
  \begin{center}
   \input{gmu.eepic}
  \end{center}
 \fi
\else
\fi
 \caption{
  \label{news.fig.article-flow}
  Usenet news flow through Groucho Marx University.
 }
\end{figure}


The labels on the arrows originating from \host{brewhq} may require some
explanation, though.  By default, it wants all locally generated news to
be sent to \host{groucho.edu}.  However, as \host{groucho.edu} does not
carry the \ngrp{fj} groups, there's no pointing in sending it any
messages from those groups.  Therefore, the feed from \host{brewhq} to
GMU is labelled \ngrp{all,!fj}, meaning that all groups except those
below \ngrp{fj} are sent to it.



\section{How Does Usenet Handle News?}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\label{news.algorithm}
\index{exchanging!news}
\index{news!exchanging|(}
\index{news!feeding|(}
\index{news!flooding algorithm}
\index{flooding algorithm}

Today, Usenet has grown to enormous proportions. Sites that carry
the whole of netnews usually transfer something like a paltry 
sixty megabytes a day.\footnote{
 Wait a moment: 60~Megs at 9600~bps, that's 60 million by 1200,
 that is\ldots mutter, mutter,\ldots Hey! That's 34~hours!
} Of course this requires much more than pushing around files.  So let's
take a look at the way most \unix{} systems handle Usenet news.


\index{feed, news}
News is distributed through the net by various transports. The
historical medium used to be UUCP, but today the main traffic is carried
by Internet sites. The routing algorithm used is called \emph{flooding}:
Each site maintains a number of links (\emph{news feeds}) to other sites.
Any article generated or received by the local news system is forwarded
to them, unless it has already been seen at that site, in which case it
is discarded. A site may find out about all other sites the article has
already traversed by looking at the \mhdr{Path:} header field. This
header contains a list of all systems the article has been forwarded by
in bang path notation.


%% We have to output the < and > below in math mode, because
%% some LaTeX's produce a heads-down ! and ? instead.
\index{news!message id}
\index{news!history}
To distinguish articles and recognize duplicates, Usenet articles have
to carry a message id (specified in the \mhdr{Message-Id:} header
field), which combines the posting site's name and a serial number into
``\email{$<$\var{serial}@\var{site}$>$}''. For each article processed, the
news system logs this id into a \emph{history} file against which all
newly arrived articles are checked.


\index{news!limit a feed}
\index{news!distribution}
The flow between any two sites may be limited by two criteria: for one,
an article is assigned a distribution (in the \mhdr{Distribution:}
header field) which may be used to confine it to a certain group of
sites.  On the other hand, the newsgroups exchanged may be limited by
both the sending or receiving system.  The set of newsgroups and
distributions allowed for transmission to a site are usually kept in the
\file{sys} file.


\index{news!batching}
\index{batching!news}
\index{delivering!news|(}
The sheer number of articles usually requires that improvements be made
to the above scheme. On UUCP networks, the natural thing to do is to
collect articles over a period of time, and combine them into a single
file, which is compressed and sent to the remote site. This is called
\emph{batching}.\footnote{
  The golden rule of netnews, according to Geoff Collyer: ``Thou shalt
  batch thine articles.''
}


\index{news!ihave/sendme}
An alternative technique is the \emph{ihave/sendme} protocol that
prevents duplicate articles from being transferred in the first place,
thus saving net bandwidth. Instead of putting all articles in batch
files and sending them along, only the message ids of articles are
combined into a giant ``ihave'' message and sent to the remote site.  It
reads this message, compares it to its history file, and returns the
list of articles it wants in a ``sendme'' message. Only these articles
are then sent.


Of course, ihave/sendme only makes sense if it involves two big sites
that receive news from several independent feeds each, and who poll each
other often enough for an efficient flow of news.


\index{news!NNTP}
Sites that are on the Internet generally rely on TCP/IP-based software
that uses the Network News Transfer Protocol, NNTP.\footnote{
 Described in RFC~977.
} It transfers news between feeds and provides Usenet access to single
users on remote hosts.


\index{news!pulling}
\index{news!pushing}
NNTP knows three different ways to transfer news. One is a real-time
version of ihave/sendme, also referred to as \emph{pushing} news. The
second technique is called \emph{pulling} news, in which the client
requests a list of articles in a given newsgroup or hierarchy that have
arrived at the server's site after a specified date, and chooses those
it cannot find in its history file.  The third mode is for interactive
newsreading, and allows you or your newsreader to retrieve articles from
specified newgroups, as well as post articles with incomplete header
information.

\index{delivering!news|)}
\index{news!exchanging|)}
\index{news!feeding|)}


\index{news!spool}
\index{news!active file@\file{active} file}
At each site, news are kept in a directory hierarchy below \newsspool,
each article in a separate file, and each newsgroup in a separate
directory.  The directory name is made up of the newsgroup name, with
the components being the path components. Thus,
\ngrp{comp.os.linux.misc} articles are kept in
\file{\newsspool/comp/os/linux/misc}. The articles in a newsgroup are
assigned numbers in the order they arrive. This number serves as the
file's name.  The range of numbers of articles currently online is kept
in a file called \file{active}, which at the same time serves as a list
of newsgroups known at your site.


\index{news!deleting old news}
\index{news!expiring old articles}
Since disk space is a finite resource,\footnote{
 Some people claim that Usenet is a conspiracy by modem and hard disk
 vendors.
} one has to start throwing away articles after some time. This is
called \emph{expiring}. Usually, articles from certain groups and
hierarchies are expired at a fixed number of days after they arrive.
This may be overridden by the poster by specifying a date of expiration
in the \mhdr{Expires:} field of the article header.


\index{news|)}
